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Abstract. We give a semiclassical analysis of a nonlinear eigenvalue problem 
arising from the study of the failure of analytic hypoellipticity and obtain a general 
family of hypoelliptic, but not analytic hypoelliptic operators. 

1. Introduction 
We are interested in a family of operators of the type 

(1.1) H(x, D x , A) = -A + (P(x) - A) 2 , 

where x i— ► P(x) is a polynomial of degree m. In the study of the failure of ana- 
lytic hypoellipticity, one approach consists in showing that the nonlinear eigenvalue 
problem 

(1.2) H(x, D x , X)v = 

has at least one non trivial solution (A, v) gCx iS(IR n ) (t) ^ 0). This has been used 
by many authors including Baouendi-Goulaouic, Helffer, Christ jHHHElEl, Hanges- 
Himonas jBJ, Chanillo pQ and quite recently by Chanillo-Helffer-Laptev |2j, where 
the reader can find a more extensive list of references. All the results lead to the 
formulation of a conjecture by Treves 22. giving a necessary and sufficient condition 
of analytic hypoellipticity extending [T3]. It could be a rather natural conjecture that, 
when x i-> P(x) is an homogeneous elliptic polynomial on MJ 1 of order m > 1, (jl.2|) 
has at least one non-trivial solution. This result is proved ([HUGE]) for n = 1 and for 
m > 2n when n = 2, 3 ( 2J). Our aim is to provide a semiclassical approach to this 
problem. Our analysis concerns actually a more general class of operators of the form 

p 

(1.3) ^2 D l s + (P(xi,...,x p )D Xp+1 -D Xp+2 ) 2 + (Q(x 1 ,...,x p )D Xp+1 ) 2 . 
j=i 

When specialized to the case (jl.2|) . we obtain the following 
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Theorem 1.1. . 

Let n be even. Let P be any real polynomial of degree m > 2 such that the homogeneous 
part of degree m is elliptic. Then the nonlinear eigenvalue problem has at least 
one non trivial solution. 

Its now standard corollary to analytic hypoellipticity (see for example [2]) is : 

Corollary 1.2. . 

Let P be elliptic homogeneous of degree m > 2 on MP, with an even p > 0, then the 
operator on M p+2 : 

v 

P(x, D x ) := D l 3 + ( P (^ ■ ■ ■ , x P )D Xp+1 - D Xp+2 ) 2 , 

is not analytic hypoelliptic at 0. 
Remark 1.3. . 

As we shall discuss more deeply in Section^ G. Metivier already showed in [12], that 
P(x, D x ) is not hypoanalytic in any open set. This is indeed a sum of squares with 
an odd number of linearly independent vector fields. But the corollary presented here 
and its proof give a stronger information at 0. 

Other examples in any dimension related to (|1.3|) are also given in Section These 
examples give an additional light to the general conjecture of |2*2*] . 

Recently, Chanillo-Helffer-Laptev ([2]) used Lidskii's Theorem to prove the exis- 
tence of nonlinear eigenvalues of (jl.2j) permitting to recover some known results for 
n = 1 and to give new examples in dimension n > 2. 

The proof of Theorem II .11 is based on the semiclassical analysis combined with the 
ideas of Chanillo-Helffer-Laptev (|2])- We follow closely the reduction of Chanillo- 
Helffer-Laptev which is recalled in Section |21 In this approach via Lidskii's theorem, 
the existence of a nonlinear eigenvalue of (jl.2J) is reduced to prove, for some k, the 
non vanishing of the trace of the k-th power of a linear operator T>: 

Tr V k ^ 0. 

Our approach is to introduce artificially a semiclassical parameter h in this operator 
(see Sectional). Then, the existence of a nonlinear eigenvalue of (jl.2|) is reduced to 
prove that for some k and for some h, 

Tr V(h) k ^ 0, 

where T>(h) is an /i-pseudo-differential operator. In other words, one can say that, 
while Chanillo-Helffer-Laptev try to prove that, for some k, 

(1.4) E A 7^ ' 

3 

where Xj are the nonlinear eigenvalues of (|1.2}1 (which can not be real under our 
assumptions), we want to find here h and k such that 

(1.5) J2( hm/{m+1)X 3 + 1 )~ k 

j 
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It is standard (see (TZj) that Tr T>(h) k has a complete semiclassical expansion when 
h — *■ 0, which is theoretically computable. The question becomes now to find a 
non-zero term in these expansions which we call the semiclassical criterion. The com- 
putation of the leading term given in Section 0] gives the complete answer for n even. 
Although we do not have a general result for all n odd, we believe that the semiclassi- 
cal criterion given below can be used to show that for each given value of n odd, (jl.2|) 
has at least one non trivial solution. The introduction of a semiclassical parameter h 
permits us to overcome the difficulty related to the non commutativity of operators 
encountered in ( 2J). 

2. Chanillo-Helffer-Laptev's approach 
We rewrite H(x, D x , A) in the form 

(2.1) H(x, D x , A) = L - 2AM + A 2 , 
with 

(2.2) L = —A + P(x) 2 , M = P(x) . 

The operator L is invertible and its inverse is a pseudo-differential operator (see |17j). 
It is also easy to give sufficient conditions for determining whether the operator 

(2.3) A := L~ l 

belongs to a given Schatten class (see the appendix in 0). Then the initial problem 
is reduced to the spectral analysis of 

(2.4) (I - 2XB + X 2 A)u = . 
with 

(2.5) B = A 1/2 PA 1/2 

Chanillo-Helffer-Laptev ([2J used a rather standard approach to transform this 
nonlinear spectral problem to a linear ones and then apply Lidskii's Theorem to 
prove the existence of a non trivial solution. 

We first recall the reduction to the linear spectral problem. It is easy to see that 
it is enough to show that the operator T> defined by 

2B A 1 ' 2 
-A 1 ' 2 



(2.6) V :-- 



has a non zero eigenvalue /i. The first component of the corresponding eigenvector is 
an eigenvector of the problem (jl.2|) with // = j-. 

If B and A are compact, T> is compact but the main difficulty is that T> is not 
selfadjoint. The Lidskii Theorem says that 

Theorem 2.1. . 

Let C be a trace class operator. Let Xj{C) denote the sequence of non zero eigenvalues 
of C. Then 

5>,-(C)= TrC. 



4 BERNARD HELFFER, DIDIER ROBERT, AND XUE PING WANG 

Here the eigenvalues are repeated according to their algebraic multipicity. 

By Lidskii's Theorem, if Tr C 7^ 0, C has at least one non zero eigenvalue. 
Corollary 2.2. (Chanillo-Helffer-Laptev 0). 

Let T> be a compact operator. Assume that there exists k 6 N* such that T> k is of 
trace class and Tr T> k ^ 0. Then il.ty) has at least one non trivial solution. 

Chanillo-Helffer-Laptev used this criterion in the case k = 2,3,4 and obtained a 
family of interesting results in non analytic hypoellipticity. The non commutativity 
between A and B makes it hard to apply their method with higher rang criteria 
(although some results can be obtained in the same vein for k — 6, 8). 

3. The semiclassical criterion 

As explained in the introduction, our goal is to introduce a semiclassical parameter 
h in the operators A and B and to apply the theory of /z-pseudo-differential operators 
to give a complete semiclassical asymptotic expansion of the trace T>(h) k . We are then 
led to find conditions under which the leading term is non zero. 

Initially, the family of operators to study is of the form 

(3.1) H(x, D x , A) = -A + (P(x) - A) 2 , 

where x ^ P(x) is a real polynomial on IR n of order m > 2. Write P in the form 

P — Pm + P-m-l + - - - + Pq j 

where Pj is a homogeneous polynomial of degree j. We assume that P m is elliptic on 

R n : 

(3.2) P m (x) ^ 0, Vx ^ 0. 

To be definite, we assume P m (x) > 0, Vx ^ 0. The case P m (x) < 0, Wx ^ is similar. 
This condition on P m requires m to be even and thus excludes polynomials of odd 
degree when n = 1. 

To introduce the semiclassical parameter h, we make the dilation x — > rx and set 
A = (A' - l)r m (One takes A = (A' + l)r m if P m is negative). Let 

(3.3) H(x, hD; A', h) = -h 2 A + (P(x, h) - A') 2 , 
where h = and 

(3.4) P{x,h) = (P m (x) + l) + /i 1 /( m+1 )p m _ 1 (x) + /i 2 /( m+1 )p m _ 2 (x) + --- + /i m /( m+1 )p • 
Then, the initial problem 

(3.5) H(x, D- \)v = 
has a non trivial solution (A, v ) if and only if 

(3.6) H(x, hD; A', h)u = 

has a non trivial solution (X',u) for some h > 0. Let us remark that 

-h 2 A + (P m + l) 2 > 1. 
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One can then prove that for h > small enough, —h 2 A + P(x, h) 2 > 1/2 and there- 
fore is invertible. 

More generally, let us consider the nonlinear eigenvalue problem 

(3.7) (-h 2 A + (Q(x, h)) 2 + (P(x, h) - z) 2 )u = 0, 
where P(x, h) is of the above form and 

(3.8) Q(x, h) = Q m {x) + h^ m+ ^Q m ^(x) + /i 2/(m+1) g m - 2 (x) + ■ • ■ + h m ^ m+ ^Q , 

with Qj a homogenous polynomial of degree j. This kind of operators appears in the 
study of non analytic hypoellipticity for operators of the form (jl.3|) . We assume that 
P m and Q m are real and there exists C > such that 

(3.9) C- 1 < x > 2m < (P m (x) + l) 2 + Q m (x) 2 < C < x > 2m , x G R n . 

Here < x >= (1 + |x| 2 ) 1,/2 . In the case Q m = 0, P m has to be an elliptic polynomial 
of degree m so that ()3.9|) is satisfied. We can show by the argument used above that 
—h 2 A + P(x,h) 2 + Q(x,h) 2 is invertible for h > small. We define the operators 
A = A(h) and B = B(h) in this new setting by : 

(3.10) A(h) = (~h 2 A + P(x, h) 2 + Q(x, hf)~\ B(h) = A(h) l/2 P(x, h)A(h) 1/2 . 

For a temperate symbol, possibly /i-dependent, K{x, £, h), we denote by K(x, hD, h) 
the /z-pseudo-differential operators defined by Weyl quantization 

(3.11) K(x, hD, h)u(x) = -i— [ [ e ^-y^ h K(^±y,^, h)u{y) dyd£ , 



(2tt/i)' 

V u G iS(lR n ). By the /i-pseudo-differential calculus [T7j, we deduce easily that A{h) 
and B{h) are /i-pseudo-differential operators with symbols, satisfying for any iV G 
IN*, 

(m+l)N 

(3.12) a(x,£;h) = hT^ aj (x, £) + h N+ ^+^R N {a, h) 

(m+l)N 

(3.13) b{x,^;h) = ^ h^b^x, £) + h N+ T^R N (b, h) . 

j=0 



Here the symbols 
(3.14) 



a = (^ + {Pm + I) 2 + Qm(x, 
bo ={e + {P m + l? + Qm{x) 2 Y\P m + l) 



2\-l 

rn \ 



are, by definition, the /i-principal symbols of A(h) and B(h), respectively. If we denote 
by the class of symbols defined as in Robert [T7] , and if we introduce : 

p. = (i + ^2 + x am)-i(i + x 2yzrf+rj(i + ^yw^>, 
<J) =(l + x 2 ) 1 / 2 , 

* =(i+a i/2 , 
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then 

% e bj e S£< x>m . 
Moreover the remainders Rn(ci, h) and Rn{o, h) are respectively a bounded family of 
symbols in S P ^ m+1)+1 and in S p ^ +1<x>m . 

The Chanillo-Helffer-Laptev's /c-criterion becomes now the following 

Lemma 3.1. . 

Let T>(h) be defined as in H2.6}) with A, B replaced by A(h), B(h). Let us assume 
there exists k such that V{h) k is an operator of trace class and that 

(3.15) TrD(/i)V0 

for some h > 0. Then the nonlinear spectral problem \3. l\j has at least one non trivial 
solution. 

To apply this lemma, we prove the following 
Theorem 3.2. . 

Let us assume the condition hS. 9\) for P and Q with m > 1. Let k > n(m + l)/m, 
n > 1. Then T>(h) k is an operator of trace class for all h > sufficiently small. We 
have, for any N, the following asymptotic expansion 

N(m+1) 

(3.16) Tr V(h) k = (2irh)- n { hJ ^ H r,n,k + 0(h N+ ^)}, 

j=o 

when h — > 0. Here Hj. n> k is independent of h and N, and can be computed from the 
symbol ofT>(h) k . In particular, 

(3.17) # 0; n,fc = / tr (<7 fc (x,£)) dxd£, 

JR 2n 

ak being the h-principal symbol ofT>(h) k . 
Proof. 

We note that A(h) 1 ^ 2 and B{h) are /i-pseudo-differential operators with symbol in 

1/2 

S£ v . Therefore V(h) k is an /i-pseudo-differential operator with matrix- valued symbol 

k 

in the class . Since m > 1, we have : 

pi e L x (M 2n ) if k > n(m + l)/m . 

Consequently, T>{h) k is a trace class operator when k > n(m + l)/m. If we denote 
by (Tk(x, £;h) the total symbol of V(h) k , it has a complete semiclassical expansion 
similar to a(h) beginning with ak, the /i-principal symbol of T>(h) k . The semiclassical 
expansion of the trace follows from the formula 

Tr V{h) k = (2nh)- n J J tr (<r k (x,£;h)) dxd£. 

Here tr denotes the trace of 2 x 2 matrices. | 

A consequence of Lemma 13.11 and Theorem 13.21 is the following 
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Corollary 3.3. (The semiclassical criterion). 

Let \3. jjj) be satisfied. If there exist fceN with k > (m + l)n/m and j G N suc/i t/iat 
Hj-,n,k 7^ ; £/ien i/ie nonlinear eigenvalue problem \3. 7| ) /ias at /east one non trivial 
solution (z, u) for each h > sufficiently small. 

4. An application of THE SEMICLASSICAL CRITERION: the classical 

CRITERION 

In this section, we apply the semiclassical criterion at the classical level, that is for 
j = 0. 

Proposition 4.1. . 

Assume that Q m = and P m > is elliptic. Let k > (m + l)n/m. Then one has 

(4.1) # 0; n,fc = 0, ifn is odd, 

(n — IV 

(4.2) H,„, t = 2( _ 1 ) U _l_J__C(P m ), = 

i/ere C n zs £/ie volume of S 11 " 1 and 

C(P m ) = [ (P m (x) + l)- k dx>0. 

In particular, we observe that : 

(4.3) Ho-,n,k 7^ when k > (m + l)n/m 
for all n even. As a consequence, we get Theorem ll.il 



Proof of Proposition 14. 11 

The condition (|3.9jl is satisfied. We compute 



(4.4) / / tr (a k {x,0) dxd£ 




where a k is the /i-principal symbol of T>{h) k . Although the symbolic calculus for 
matrix-valued /z-pseudo-differential operators is complicated, the /i-principal symbol 
of T>(h) k can be easily computed. Since the /i-principal symbol of T>(h) is 



1/2 



2b a 
—a Q ' 



the /i-principal symbol of V{h) is 

I 2b n a n ' z \ 

Ok 



1/2 n 



Therefore, 



(4.5) tr a k = (b + Jb 2 -a ) k + (6 - \ b 2 - a ) k . 
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We recall that 

a = (e + (P m + I) 2 )" 1 , b = (e + (P w + lf)-\P m + 1). 
By the change of variables £ — > (P m + l)r], we obtain that 

(4.6) ! [ tr (a k )(x,Z)dxdZ = C(P m ) [ (1 + r?)- k {{l + i\ V \) k + (1 - t\ V \) k ) d V . 



This shows that 



/■oo 

(4.7) H . nik = 2C n C(P m )$t / (1 + r 2 )" fc (l + irfr^ 1 dr, k > (m + l)n/m, 

Jo 

C n being the volume of S 71 ^ 1 . 



Let 



oo 



L(n,k)= (l + r 2 )- k (l+tr) k r n - 1 dr = (1 - ^V" 1 dr. 
Jo Jo 

An integration by parts gives : 

n — 1 

(4.8) L(n,k) = i- L(n-l,k-l) 

k — 1 

„ o (n- l)(n - 2) ■ • - 2 

= i -r, w, \ A tL(2, k - n + 2) . 

(k - l)(k — 2) ■ ■ ■ (k - n + 2) v ; 

Since 

L(2 ' j) = "(j-l)(j-2)' 

we have 

(4.9) H, n , t = 2C n W' (k _ m _ 2) ( "-^ - + m _ n) }C(P„). 
This proves flUU) and (JOj). I 

Using the classical criterion, we can construct a family of examples when n is odd. 
Corollary 4.2. . 

Let n = ni + n,2 with n% even and > 1. £e£ P, R be real elliptic homogeneous 
polynomials 1 of degree m onW 11 andW 12 , respectively. For x G W 1 , set x = (x',x") G 
R ni x R n2 . Then, the operator : 

-A x + (R(x")D yi f + (P(x')D yi - D 2 



is not analytic hypoelliptic at in 



yij 1 v- 1 \^ j-^yi ^V2) i 

n+2 



Proof. 

It suffices to show that the nonlinear eigenvalue problem 

' n \ 

Y^Dl j + (R(x")) 2 + (P(x')-z) 2 ju = 



lr The important property is only that : — A x » + R(x") 2 on R™ 2 has an eigenvalue Aq. 
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has a non trivial solution (z, u). We can look for v(x') satisfying, for some z, 

(-A x t + A + (P(x') - z'f\ v = 

where Ao > is an eigenvalue of — A x « + R(x") 2 . The corresponding semiclassical 
operator is -h 2 A x , + A /i 2m/(m+1) + (P(x') — z) on W 11 . We can apply Proposition 
14.11 for Q = Q = Ao to conclude. | 

Remark 4.3. . 

One can play with Ao- We can indeed consider a sequence X n of eigenvalues of —A x n + 
R(x") 2 tending to +oo and to choose a corresponding sequence h n of h's tending to 

2m 

; such that X n hn +1 = 1. This could permit to relax the condition of ellipticity of P, 
particularly if we add a Q like in \3. 7\ ). 

Let us now consider the more general case Q m ^ 0. For (P m , Q m ) satisfying ()3.9|) . 
we can check that the classical criterion still works in some cases. Let us remark that 

(4.10) tr a k = (b + ^Jb 2 -a ) k + (b - ^Jb 2 -a ) k 
with now 

a = (£ 2 + {Pm + I) 2 + QlT\ b = (e + (P m + l) 2 + Q 2 m y\P m + 1). 

Let 

(4.11) T =((P m + l) 2 + Q 2 J 1 / 2 , n = (P n + l)/T, r 2 = \Q m \/T. 
By the change of variables £ — > Tr], we obtain that 

(4.12) / tr(a k )(x,0^ = 2T n ~ k ^[ (1 + rfY^n + i{r 2 2 + ri 2 ) 1 ' 2 )' dr,. 
The case n — 1. . 

This case can be also thoroughly analyzed by the method of [2|. When n — 1, we 
obtain here for k = 3, 

(4.13) # 0; i,3 = —-7T [ nrlT- 2 dx. 

It is clear that if P m > (this requires m to be even) and Q m ^ 0, then, t\ > and 
-f^o ; i,3 < 0. If P m changes sign, Po;i,3 may be vanishing. 

The case n > 2. . 

When n > 2, k > (m + l)n/m, 

(4.14) P 0; n, fc = 2CJ£ / T(x) n ~ fc f°(l + r 2 )- fc (n + t(r 2 + r 2 ) 1 / 2 ) V" 1 dr. 
Set 

/■oo 

(4.15) C nyk = / (l + r^-^n + ^^ + r 2 ) 1 / 2 )^- 1 ^. 
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By a change of variable, we obtain 



( Tl _ it )-fc( f 2_ r 2 ) (n-2)/2 t df 



The subcase n 
In the case n = 




2. . 



k-2 



Ti(ri + ir 2 
k-l 



k-l 



(4.16) 




1, we can see 



that 



7i(l + 2rf)/2 > 0, if n > 



(4.18) 



5^2,4 




This shows Hq._ 2 $ > for n = 2 when P m > 0. The classical criterion can be used in 
this case for m > 3 when n = 2. 

To study the case n = 2 and m = 2, we need information on the sign of Lf 0: 2,4 which 
depends on the relation between P 2 and Q 2 . An elementary computation gives : 

Lemma 4.4. . 

(1) IfTi(xi,x 2 ) 2 < (3 - \/3)/4 /or a// (xi,x 2 ), #ien i^ 0; 2,4 > 0. 

(2) IfT 1 (x 1 ,x 2 ) 2 > (3 - \/3)/4 /or all {xi,x 2 ), then H . 2A < 0. 

The subcase n — 3. . 
When n = 3, /c = 5, we have 



so long as Q m is not identically zero and P m > 0. 

Some applications. . 
We obtain the following : 

Proposition 4.5. . 

(a). Let n = 1. The operator 



is not analytic hypoelliptic at G M 3 /or all m > 2 even and cGl. 

('ft). Lei n = 2. Lei P ; Q be real homogenous polynomials of degree m > 2 satisfying 
\3. ty) and P > 0. Whenm = 2, we assume in addition that one of the above conditions 
on T\ is satisfied. Then the operator 



(4.20) D 2 X + D 2 y + (D s - P(x, y)D t ) 2 + (Q(x, y)D t ) 2 , (x, y) G M 3 , 



(4.19) 




P m (x, D) := D 2 X + (D s - cx m D t ) 2 + (x m D t ) 2 



is not analytic hypoelliptic at in M 4 . 
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(c). Let n — 3. Let P, Q be real homogenous polynomials on M. 3 of degree m > 2 
satisfying \3. 9\) . Assume that Q ^ 0,P > 0. Then the operator 

(4.21) -A x + (D s - P(x)D t ) 2 + (Q(x)D t ) 2 

is not analytic hypoelliptic at in M. 5 . 

We believe that the condition on T\ in (b) is technical. As examples of (P, Q) 
satisfying the conditions of (b) of Proposition 14.51 we can take P(x, y) = (x 2 + y 2 ) e , 
Q(x,y) = (xy) £ , V £ > 1, because in the case £ — 1, one can check that Ti(x,y) 2 > 
4/5 > (3 - y/3)/A. We shall come back at the analysis of this example corresponding 
to i = 1 in the next section. 

Remark 4.6. . 

Let us briefly compare the results obtained here and those of [2J. Apparently, in the 
case Q = 0, the "classical" criterion does not produce any result for n > 1 odd. But 
the semiclassical criteria may still work by considering a coefficient of higher order 
of the expansion of the trace. As indicated by J. Sjdstrand, a similar approach is 
used by L. Nedelec in [03 for getting a lower bound for the number of resonances of 
an h-pseudodifferential system. The same condition on the dimension appears. The 
"quantum" criterion given in [2] works forn = 1, 2, 3 but with a stronger condition on 
m when n > 1. Moreover it has been observed in Remark 4-4 ^ n that the condition 
of ellipticity of P can be replaced by a weaker condition. The last point is that the 
homogeneity of P plays an important role in the dilation argument of 0, while in 
the semiclassical approach, the lower order parts can be included. This appears to be 
useful in the dimension reduction. 

5. Comparison with Metivier's results 

In this section, we would like to analyze the links between our results and the 
previous results obtained by G. Metivier [TTJ [T3J [H] . 

5.1. A first family of examples. Let us first consider the operator on W n+2 : 

H(X, D x ) = -A + (P(x)D Xn+1 - D Xn+2 ) 2 , 

where X = (x, x n+ i, x n+2 ), P is an homogeneous positive elliptic polynomial of de- 
gree m > 2 on R n . 

If we take the "microlocal spirit" we observe that H is an operator with double 
characteristics, whose principal symbol is the function : 

(T*M"+ 2 \ 0) 9 (X, S) » K| 2 + {P(x)Ui - tn + 2? ■ 

The symbol vanishes exactly at order 2 on the submanifold 

E = {(A, S) | £ = , P(x)£ n+1 - £ n+2 = , £ n+1 + 0} . 

This submanifold is of codimension n+ 1. Let us now analyze the "symplecticity" of 
E. We recall that E is said to be symplectic if the restriction to E of the canonical 
2-form is non degenerate. An easy way for verifying the symplecticity is to con- 
sider the (n + 1) x (n + 1) matrix Ui,Uj where Ui(X, H) = £j for i = 1, • • • , n and 
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u n+ x(X, 5) = P(x)£ n+ i — £ n+2 an d to show that it is not degenerate. An immediate 
computation shows that its rank at a given point is 2 if VP ^ and if VP = 0. 

When P is elliptic and homogeneous, we get that the rank is constant outside 
and equal to 2. There are two cases : 

(1) When n — 1, we get that S is symplectic except at the points of E such that 
x = 0. The result of Treves- Tartakoff-Metivier-Sjostrand El El ES] shows 
that the operator is microlocally analytic hypoelliptic outside of E (ellipticity) 
and in the neighborhood of the points of E such that x ^ 0. In this case, the 
operator is not analytic hypoelliptic at any point (0, x n+ i, x n+ 2). 

(2) When n > 1, Metivier's result 2 PU HSJ gives that the operator H is not 
analytic hypoelliptic in any open set in M n+2 . What we show here is the more 
precise result that P is not analytic hypoelliptic at any point (0, x n+ i, x n+2 ) 
which is a finer property. See the introduction of ^3] for the discussion between 
the definitions of analytic hypoellipticity and germ-hypoanalyticity (analytic 
hypoellipticity in a neighborhood of a point and analytic hypoellipticity at a 
point). 

5.2. A new class of non analytic hypoelliptic operators. We now show that 
maybe more interesting examples can be treated when considering the more general 
class : 

H(X, D x ) = -A + (P(x)D Xn+1 - D Xn+2 f + Q(x) 2 D 2 Xn+i , 

where P and Q are homogeneous polynomials of degree m > 1 with P > and 
P 2 + Q 2 elliptic. When restricting H to x n+ \ independent distributions, we get an 
analytic hypoelliptic operator on M n+1 : 

-A x + (p(x) 2 + g(x) 2 )p 2 n+2 , 

by applying a theorem of Grushin (0). 

We have seen in Section 01 that the "classical" criterion can give a result under 
some additional condition. We will concentrate our analysis to the specific case when : 

n = 2 , P(x) = x\ + x\ , Q[x) = ax\X 2 , 

with a > 0. 

Let us do the same microlocal analysis as in the previous subsection. The charac- 
teristic set S is now defined as a union of two regular submanifolds of dimension 4 
in R 8 \ : 

E = Si U S 2 , 

with 

^ = {6 = o , 6 = o , (xl + x 2 )£ 3 , x 3 = 0,6 7^ 0} . 
Moreover Ej is symplectic outside Ei fl E 2 and not symplectic at Ei n E 2 . 



2 Note that the operator is hypoelliptic with loss of one derivative in R™+ 2 \ {x = 0}. 
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Outside Si D E 2 , observing that the symbol of # vanishes exactly at order 2 on E 
there, we get again from [2UJ HH EEZI EH] that # is microlocally analytic hypoelliptic. 
Metivier's criterion of non-analytic-hypoellipticity can not be applied at the points 
(0,0, £3,0:4) (the operator is indeed not hypoelliptic with loss of one derivative) and 
it is interesting to see what is obtained through our approach. 

Proposition 5.1. . 

For a > small enough, the operator H(X, Dx) is not analytic hypoelliptic at any 
point (0, x 3 , x 4 ). 

Proof. 

We just apply the criterion of the previous section (Proposition 031 second case) and 
the discussion following the statement. 

When a is small enough (at least < a < 1), we observe that the quantity 
(P m + l) 2 / ((P m + l) 2 + Qm) i s sufficiently near 1. In particular, the second condition 
on T\ in Lemma [4.41 is satisfied. 

Remark 5.2. . 

As explained to us by M. Christ, it is possible to prove, for rather large classes of 
models depending analytically on an additional parameter a, that some associated 
Fredholm determinant is analytic in a. One can conclude that, if the operator is not 
analytic hypoelliptic for some value of a, then it is not analytic hypoelliptic for generic 
values of a. We refer to j5J 7 Proposition 5.2 for the argument. In the particular case 
of the above Proposition ^. 11 we can present the argument in the following way. Let 
#0,2,4(0) denote #0,2,4 defined as in Theorem YS .2\ with P = x\ + x\ and Q = axix 2 . 
Using \4-ltf) an d (|^--?6| ), one can check that #0,2,4(0;) is real analytic in a > 0. The 
proof of Proposition ^. 1\ shows that #0,2,4(0) 7^ for a > small. Thus, #o ; 2,4(o;) 7^ 
for all a > 0, except a discrete set in K + and therefore Proposition ^. 1\ remains true 
in this case. This argument can be used for more general cases. It is indeed easy to 
check in many situations the analyticity of #o, n ,fc with respect to the parameter. 
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